Interpreting Expressions Having Potentially Ambiguous Meanings In Different Domains

ABSTRACT

The present invention extends to methods, systems, and computer program products for interpreting expressions having potentially ambiguous meanings in different domains. Multi-domain natural language understanding systems can support a variety of different types of clients. Expressions can be interpreted across multiple domains. Weights can be assigned to domains. Weights can be client specific or expression specific so that a chosen interpretation is more likely correct for the type of client or for its context. Stored weight sets can be chosen according to identifying information carried as metadata with expressions or weight sets carried directly as metadata. Domains can additionally or alternatively be ranked in ordered lists or comparative domain pairs of to favor some domains over others as appropriate for client type or client context.

CROSS-REFERENCE TO RELATED APPLICATIONS

Not applicable.

BACKGROUND 1. Field of the Invention

This invention relates generally to virtual assistants, and, moreparticularly, to virtual assistants configured to understand naturallanguage.

2. Related Art

Modern virtual assistants can answer questions and carry out commandsexpressed using natural language. More advanced virtual assistants canhandle questions and commands in many domains. Domains are distinct setsof related capabilities, such as providing information relevant to aparticular field or performing actions relevant to a specific device.For example, some virtual assistants can give the weather forecast,answer questions with facts from Wikipedia, play requested music, playrequested videos, send short message service (SMS) messages, make phonecalls, etc. Different virtual assistants can be developed to handlequestions and commands in different combinations of domains.

In some environments, virtual assistants run on a server that supportsmultiple different types of clients, such as, for example, smartspeakers, mobile phones, automobiles, vending machines, etc. Eachdifferent type of client can provide a virtual assistant configured tohandle questions and commands across a set of domains. The number ofdomains can vary from as few as a single domain for very specializedvirtual assistants to multiple domains for broadly useful virtualassistants. Domains may be unique to a virtual assistant or shared amongmultiple virtual assistants. For example, many virtual assistants canuse a common weather domain. On the other hand, a virtual assistant fora retailer may have a unique domain to answer queries about items instock.

At times, a virtual assistant can receive a natural language expressionthat is potentially ambiguous. That is, the expression potentially makessense in more than one domain. For example, a user uttering the verbalexpression “play footloose”, could refer to a popular song in a musicdomain and a popular movie in a video domain. The expression, “

” could refer to a popular song in a music domain and a popular movie ina video domain. For another example, the expression, “how high is mounteverest” (“

”) could refer to a height in a geography domain or a temperature in aweather domain.

Upon receiving an expression, a virtual assistant can compute a scorefor each of a plurality of different domains that roughly indicates howwell the expression makes sense in that domain. The virtual assistantchooses the domain with the best score and uses interpretation from thatdomain to produce a response for the user. For potentially ambiguousexpressions, multiple domains may provide high scores. However, the best(highest) scoring domain may not be the domain that the user intendedprocess the potentially ambiguous expression. As a result, the virtualassistant gives an inappropriate response, which frustrates the user.

BRIEF DESCRIPTION OF THE DRAWINGS

The specific features, aspects, and advantages of the present inventionwill become better understood with regard to the following descriptionand accompanying drawings where:

FIG. 1 illustrates an example computer environment for interpreting anexpression.

FIG. 2 illustrates an example of interpreting an expression at amulti-domain natural language understanding (NLU) interpreting server.

FIG. 3 illustrates another example of interpreting an expression at amulti-domain natural language understanding (NLU) interpreting server.

FIG. 4A illustrates an example expression package.

FIG. 4B illustrates another example expression package.

FIG. 5A illustrates an additional expression package.

FIG. 5B illustrates a further expression package.

FIG. 6 illustrates an example of interpreting audio samples at amulti-domain natural language understanding (NLU) interpreting server.

FIG. 7 illustrations an example of generating a response to anexpression at a multi-domain natural language understanding (NLU)interpreting server.

FIG. 8 illustrates an example Graphical User Interface (GUI)configurator screen.

FIG. 9 illustrates an example of using domain ranks to select a domainat a multi-domain natural language understanding (NLU) interpretingserver.

FIG. 10A illustrates an example rotating disk.

FIG. 10B illustrates an example Random Access Memory (RAM) chip.

FIG. 11A illustrates an example solder ball side of a packaged System OnChip (SoC).

FIG. 11B illustrates an example architecture of a System On Chip (SoC).

FIG. 12A illustrates an example of a racked based server.

FIG. 12B illustrates an example architecture of a server.

DETAILED DESCRIPTION

The present invention extends to methods, systems, machines, manufactureproducts, and computer program products for interpreting expressionshaving potentially ambiguous meanings in different domains

Aspects of the invention can weight and/or prioritize differentinterpretations of an expression provided by different domains.Weighting and/or prioritizing can be used to select a more appropriateinterpretation of an expression for a user. Weighting and/orprioritizing can thus allow developers to develop virtual assistants forspecific uses and for use in specific conditions to make the virtualassistants more likely to choose appropriate (correct) interpretationsbased on user intent. Weighting and/or prioritizing can also provide anadvantage of allowing developers to decrease the likelihood of, andpotentially prevent, users from getting results from competitors'domains.

In some aspects, an interpreter is configured to handle expressionsacross a plurality of different domains. The interpreter receives anexpression from a user. The expression can be text (e.g., AmericanStandard Code for Information Interchange (ASCII) characters or Unicodecharacters), tokenized sequences of morphemes, speech audio samples,etc. entered by a user. Speech recognition can be used to extractsequences of morphemes from speech audio samples.

The interpreter interprets the expression in each domain to compute afirst likelihood score for the expression in each domain. Theinterpreter uses metadata associated with the expression to select acorresponding weight for each domain. In some aspects, metadata includesone or more of: a client ID, a vendor ID, a product ID, a version, auser ID, a location, sensor information, etc. In these aspects, aninterpreter selects weights based on contents of the metadata. In otheraspects, weights are included in metadata. In these other aspects, themetadata can specify a domain name or domain ID associated with eachweight. Weights can be represented as integers, floating point numbers,or other symbolic representations.

For each domain, the interpreter applies the corresponding weight to thefirst likelihood score to compute a second likelihood score. The secondlikelihood scores are useful for selecting a domain, from among theplurality of domains, to provide an interpretation of the expression.For example, a domain having a second score exceeding a threshold can beselected as the domain to interpret the expression. A selected domaincan be used to determine user intent and compute an appropriateresponse. Responses can be visual or audible (e.g., a sound or spokeninformation) and can cause a device to perform an operation, such assending text or audible messages, etc.

Developers can develop interpreters (e.g., using a platform with agraphical user interface) that include different domains from among aplurality of available domains and can assign a weight to each includeddomain.

Alternately or in combination with the use of weights, aspects of theinvention can also utilize domain ranks. If multiple domains have ascore exceeding a threshold, a higher ranked domain can be selected asthe domain to interpret an expression. Thresholds can be the same ordifferent among different domains. Ranks can be specified as an orderedlist, as pairs of domains with preferences, based on context (e.g., whatapplications are running, recent preceding dialog subjects, etc.). Inone aspect, a rank is specified for “other” domains that are nototherwise expressly ranked.

Aspects include cloud-based multi-domain Natural Language Understanding(NLU) interpreting platforms that support multiple client types. Clientdevelopers can develop clients that use other's domains and/or candevelop customer or proprietary domains for their clients. For example,a car maker might develop a proprietary car control domain that adjuststhe heater and windows for their cars. The car maker can also develop aclient that uses the proprietary car control domain as well as othercommonly used domains, such as, domains that answer weather forecastqueries. A client developer can assign higher weights to their owncustom or propriety domains.

A client developer may also have a business relationship with a domaindeveloper. The client developer can assign higher weights to domainsfrom the domain developer or can assign lower weights (or even a zeroweight) to domains from competitors of the domain developer.Accordingly, the client developer can essentially prevent the clientfrom selecting competitor domains for responding to expressions orallowing competitor domains to respond to expressions of littlerelevance to the client

For example, a grocery shopping client from a grocery store company mayhave a custom domain that includes information specific to theirproducts. Using weights, the grocery shopping client can be configuredto respond to food-related queries using the custom domain for productshaving a store brand. For example, the customer domain can be selectedto respond to a query “how much energy is in a doughnut” with specificinformation about their store brand of doughnuts. The custom domain canbe selected instead of a domain for a brand name doughnut company. Theclient can respond to food-related queries for products not having astore brand with a domain for general nutritional information. Forexample, a common domain can be selected to respond to a query “how manycalories in a banana” when the store doesn't have a store brand forbananas.

Aspects of the present invention provide certain advantages that areimprovements over conventional approaches. Specifically, ambiguousexpressions are interpreted with a greater probability of matching auser's intent. Certain expressions can be understood that otherwisewouldn't. The behavior of devices is optimized for their type and theirsituation. Users are better satisfied with the natural languageinterfaces of devices. Shared cloud-based interpretation servers canprovide different levels of service to different device developers (andcharge different pricing levels). Furthermore, more accurateinterpretation of natural language expressions is valuable fordecreasing error rates in safety-critical systems. Other advantages ofthe present invention will be apparent to practitioners in the art.

Embodiments of the present invention may comprise or utilize a specialpurpose or general-purpose computer including computer hardware, suchas, for example, one or more processors and system memory, as discussedin greater detail below. Embodiments within the scope of the presentinvention also include physical and other computer-readable media forcarrying or storing computer-executable instructions and/or datastructures. Such computer-readable media can be any available media thatcan be accessed by a general purpose or special purpose computer system.Computer-readable media that store computer-executable instructions arecomputer storage media (devices). Computer-readable media that carrycomputer-executable instructions are transmission media. Thus, by way ofexample, and not limitation, embodiments of the invention can compriseat least two distinctly different kinds of computer-readable media:computer storage media (devices) and transmission media.

Computer storage media (devices) includes random access memory (RAM),read-only memory (ROM), electrically erasable programmable read-onlymemory (EEPROM), compact disc read-only memory (CD-ROM), solid statedrives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory(“PCM”), other types of memory, other optical disk storage, magneticdisk storage or other magnetic storage devices, or any other mediumwhich can be used to store desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer.

A “network” is defined as one or more data links that enable thetransport of electronic data between computer systems and/or modulesand/or other electronic devices. When information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or a combination of hardwired or wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Transmission media can include a network and/or data links whichcan be used to carry desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer. Combinationsof the above should also be included within the scope ofcomputer-readable media.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media to computerstorage media (devices) (or vice versa). For example,computer-executable instructions or data structures received over anetwork or data link can be buffered in RAM within a network interfacemodule (e.g., a “NIC”), and then eventually transferred to computersystem RAM and/or to less volatile computer storage media (devices) at acomputer system. RAM can also include solid state drives (SSDs orPeripheral Component Interconnect eXtended (PCI-X) based real-timememory tiered Storage, such as FusionIO). Thus, it should be understoodthat computer storage media (devices) can be included in computer systemcomponents that also (or even primarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at a processor, cause a general purposecomputer, special purpose computer, or special purpose processing deviceto perform a certain function or group of functions. Thecomputer-executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the invention may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,personal digital assistants (PDAs), tablets, pagers, routers, switches,various storage devices, and the like. The invention may also bepracticed in distributed system environments where local and remotecomputer systems, which are linked (either by hardwired data links,wireless data links, or by a combination of hardwired and wireless datalinks) through a network, both perform tasks. In a distributed systemenvironment, program modules may be located in both local and remotememory storage devices.

Embodiments of the invention can also be implemented in cloud computingenvironments. In this description and the following claims, “cloudcomputing” is defined as a model for enabling ubiquitous, convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, servers, storage, applications, and services)that can be rapidly provisioned via virtualization and released withminimal management effort or service provider interaction, and thenscaled accordingly. A cloud model can be composed of variouscharacteristics (e.g., on-demand self-service, broad network access,resource pooling, rapid elasticity, measured service, etc.), servicemodels (e.g., Software as a Service (SaaS), Platform as a Service(PaaS), Infrastructure as a Service (IaaS), and deployment models (e.g.,private cloud, community cloud, public cloud, hybrid cloud, etc.).Databases and servers described with respect to the present inventioncan be included in a cloud model.

Further, where appropriate, functions described herein can be performedin one or more of hardware, software, firmware, digital components, oranalog components. For example, one or more application specificintegrated circuits (ASICs) can be manufactured or field programmablegate arrays (FPGAs) programmed to carry out one or more of the systemsand procedures described herein. Certain terms are used throughout thefollowing description and Claims to refer to particular systemcomponents. As one skilled in the art will appreciate, components may bereferred to by different names. This document does not intend todistinguish between components that differ in name, but not function.

FIG. 1 illustrates a computer environment 100 for interpreting anexpression. More specifically, computer environment 100 depicts userinteraction with a client connected to a cloud-based multi-domainnatural language understanding (“NLU”) interpreting server.

User 10 can speak verbal expression 11 within sensor range of smartspeaker 12. Smart speaker 12 can receive verbal expression 11 and sendverbal expression 11 over a network to multi-domain NLU interpretingserver 13 in cloud 16. Mobile phone client 14 and automobile client 15can also send natural language expressions to multi-domain NLUinterpreting server 13.

As depicted, verbal expression 11 can be the phrase “play footloose” “

”. The phrase “play footloose” “

” can indicate both the name of a popular video and the name of apopular song. As such, the phrase “play footloose” “

” may receive high scores for interpretation in both a song domain and avideo domain. However, since smart speaker 12 has no visual display,higher weight can be given to the song domain relative to the videodomain. Thus, the phrase “play footloose” “

” received from smart speaker 12 can be interpreted as a request to playa song.

On the other hand, a client with a display, such as mobile phone 14,might give a higher weight to a video domain than to a song domain.Thus, the phrase “play footloose” “

” received from mobile phone 14 can be interpreted as a request to playa video.

Some clients, such as mobile phone 14 or automobile 15, can rundifferent kinds of applications. On these clients, running applicationscan be used to determine relative domain weights that favor songs orvideos. For example, if mobile phone 14 has a music application in theforeground, domain weights can be configured to favor playing a song.

Stored Domain Weights

FIG. 2 illustrates a more detailed view of a multi-domain naturallanguage understanding (NLU) interpreting server 13. Multi-domainnatural language understanding (NLU) interpreting server 13 can receiveexpression package 5 from a client device, such as, for example, fromany of smart speaker 12, mobile phone 14, or automobile 15. Multi-domainnatural language understanding (NLU) interpreting server 13 forwardsexpression 19 to interpreter 20 and forwards client ID 31 to weightselector 23. Interpreter 20 interprets expression 19 according to eachof domains 21 (e.g., domain 21A, domain 21B . . . domain 21N) to produceinterpretations 32 1-N. In some aspects, interpreter 20 producesinterpretations for less than all of domains 21. Interpreter 20 alsoproduces first scores 33 1-N. Each of first scores 33 corresponds to oneof interpretations 32. In some aspects, scoring is performed prior tocompleting interpretations of expression 19. Scoring prior tointerpretation can save processing resources. Interpreter 20 can sendfirst scores 33 1-N to rescore module 24.

Weight selector 23 has access to multiple sets of weights 22 (possiblystored in durable storage), including weights 22A, 22B, and 22C. Aweight set can include a weight for one or more or possibly all ofdomains 21. Weights in a weight set can be represented as floating-pointnumbers, integers, text, or as other appropriate formats.

Weight selector 23 receives client ID 31. In one aspect, the clientdevice that received expression 19, sends client ID 31 as metadata alongwith expression 19 to multi-domain natural language understanding (NLU)interpreting server 13. Multi-domain natural language understanding(NLU) interpreting server 13 separates client ID 31 from expression 19and forwards client ID 31 to weight selector 23. Client ID 31 canrepresent a client type, such as, for example, smart speaker, mobilephone, automobile, etc. Weight selector 23 selects a weight set, suchas, for example, weights 22A, corresponding to client ID 31. Weightselector 23 can send weights 22A to rescore module 24.

Other types of identifying data, such as, for example, a vendor ID, aproduct ID, a version, a user ID, a location, sensor information, etc.can also be transferred to weight selector 23. Weight selector 23 canuse any of the described types of identifying data to select a weightset.

Rescore module 24 receives first scores 33 1-N from interpreter 20 andweights 22A from weight selector 23. Rescore module 24 applies weights22A to first scores 33 1-N to produce second scores 34 1-N. Rescoringcan include multiplying a first score by a corresponding weight (or avalue 1 for scores that have no associated weight) to produce a secondscore. Other rescoring mechanisms, such as addition, adding tologarithms, threshold comparisons, filtering, and application ofarbitrary functions can also be used.

Based on second scores 34 1-N, multi-domain natural languageunderstanding (NLU) interpreting server 13 can select an appropriateinterpretation of expression 19 (e.g., an interpretation with thehighest score, an interpretation that exceeds a threshold for acorresponding domain, etc.) from among interpretations 32 1-N. Based onthe selected appropriate interpretation, multi-domain natural languageunderstanding (NLU) interpreting server 13 can generate an appropriateresponse to expression 19. An appropriate response can include causingthe client device to perform an operation, such as outputting text oraudible messages, playing a song, playing a video, moving, etc.

Receiving Domain Weights as Metadata

FIG. 3 illustrates another more detailed view of multi-domain naturallanguage understanding (NLU) interpreting server 13. Multi-domainnatural language understanding (NLU) interpreting server 13 can receiveexpression package 6 from a client device, such as, for example, fromany of smart speaker, 12, mobile phone 14, or automobile 15.Multi-domain natural language understanding (NLU) interpreting server 13forwards expression 29 to interpreter 20 and forwards weights 41 torescore module 24. Interpreter 20 interprets expression 29 according toeach of domains 21 (e.g., domain 21A, domain 21B . . . domain 21N) toproduce interpretations 42 1-N. In some aspects, interpreter 20 producesinterpretations for less than all of domains 21. Interpreter 20 alsoproduces first scores 43 1-N. Each of first scores 43 corresponds to oneof interpretations 42. In some aspects, scoring is performed prior tocompleting interpretations of expression 29. Scoring prior tointerpretation can save processing resources. Interpreter 20 can sendfirst scores 43 1-N to rescore module 24.

In one aspect, the client device that received expression 29, sends(per-expression) weights 41 as metadata along with expression 29 tomulti-domain natural language understanding (NLU) interpreting server13. Multi-domain natural language understanding (NLU) interpretingserver 13 separates weights 41 from expression 19 and forwards weights41 to rescore module 24. Weights 41 can include a weight for one or moreor possibly all of domains 21. Weights in weights 41 can be representedas floating-point numbers, integers, text, or as other appropriateformats.

Rescore module 24 receives first scores 43 1-N from interpreter 20 andweights 41 from the client device. Rescore module 24 applies weights 41to first scores 43 1-N to produce second scores 44 1-N. Rescoring caninclude multiplying a first score by a corresponding weight (or a value1 for scores that have no associated weight) to produce a second score.Other rescoring mechanisms, such as addition, adding to logarithms,threshold comparisons, filtering, and application of arbitrary functionscan also be used.

Based on second scores 44 1-N, multi-domain natural languageunderstanding (NLU) interpreting server 13 can select an appropriateinterpretation of expression 29 (e.g., an interpretation with thehighest score, an interpretation that exceeds a threshold for acorresponding domain, etc.) from among interpretations 42 1-N. Based onthe selected appropriate interpretation, multi-domain natural languageunderstanding (NLU) interpreting server 13 can generate an appropriateresponse to expression 29. An appropriate response can include causingthe client device to perform an operation, such as outputting text oraudible messages, playing a song, playing a video, moving, etc.

Aspects also include combinations of stored weights and per-expressionweights. Per-expression weights can be used to override or bias storedweights per domain for a client.

Expression Packages

FIG. 4A illustrates an example expression package 400. Expressionpackage 400 can be transferred from a client device to an NLUinterpreting server. As depicted, expression package 400 includes clientID 401 and expression data 402. Client ID 401 can be transferred priorto expression data 402 to allow the NLU interpreting server chose aweight set before receiving expression data 402. A weight selector canuse client ID 401 to select a weight set. A rescore module can applyweights in the selected weight set to first scores produced frominterpreting expression data 402 across a plurality of domains.

FIG. 4B illustrates another example expression package 450. Expressionpackage 450 can be transferred from a client device to an NLUinterpreting server. As depicted, expression package 450 includes vendorID 451, product ID 452, and expression data 453. A weight selector canuse vendor ID 451 and product ID 452 to select a weight set. A rescoremodule can apply weights in the selected weight set to first scoresproduced from interpreting expression data 453 across a plurality ofdomains.

Transferring vendor ID 451 and product ID 452 is useful for NLUinterpreting servers that support an ecosystem of different companieswith natural language clients. For example, an NLU interpreting servercan offer per-vendor accounts along with different processing functionsor software for different products and product versions from differentvendors. Various other formats, such as, for example, a version, a userID, a location, sensor information, etc. can also be appropriate forindicating ID information used to select an appropriate weigh set forexpression data.

FIG. 5A illustrates an additional expression package 500. Expressionpackage 500 can be transferred from a client device to an NLUinterpreting server. As depicted, expression package 500 includesweights 501 and expression data 502. Weights 501 can include a weightfor each of one or more domains. A rescore module can apply weights inweights 501 set to first scores produced from interpreting expressiondata 502 across a plurality of domains.

FIG. 5B illustrates a further expression package 550. Expression package550 can be transferred from a client device to an NLU interpretingserver. As depicted, expression package 550 includes weights 551A, 551B,551C, corresponding tags 552A, 552B, and 552C respectively, andexpression data 550. Each of tags 552A, 552B, and 552C indicate a domainto which a corresponding weight 551A, 551B, and 551C respectively isapplicable. A rescore module can apply weights 551A, 551B, and 551C tofirst scores for domains indicated by tags 552A, 552B, and 552Crespectively. The rescore module can apply a neutral weight (e.g., avalue of 1 for multiplicative rescoring) to any domains not expresslyindicated in expression package 550.

Additional Functionality

As described, expression data can take a variety forms including ASCIIcharacter strings, Unicode character strings, a sequence of digitalaudio samples representing speech, a sequence of gestures, and patternsof neural activity.

FIG. 6 illustrates an additional more detailed view of multi-domainnatural language understanding (NLU) interpreting server 13.Multi-domain natural language understanding (NLU) interpreting server 13can receive expression package 64 from a client device, such as, forexample, from any of smart speaker, 12, mobile phone 14, or automobile15. Multi-domain natural language understanding (NLU) interpretingserver 13 forwards audio samples 62 to speech recognition module 61 andforwards metadata 63 to weight selector 23 and/or to rescore module 24as appropriate.

Speech recognition module 61 performs speech recognition on audiosamples 62 to produce expression 66 (a text representation of audiosamples 62). Speech recognition module 61 can produce a set of multiplehypotheses of recognized sequences of speech tokens. A speechrecognition accuracy estimate score can be computed for each hypothesis.Expression 66 can include ASCII characters, Unicode characters, such asones including Chinese characters, or other lexical or ideographicalrepresentations of audio samples 62.

Speech recognition module 61 can send expression 66 to interpreter 20.Interpreter 20 can interpret expression 66 according to each of domains21 (e.g., domain 21A, domain 21B . . . domain 21N) to produceinterpretations 67 1-N. In some aspects, interpreter 20 producesinterpretations for less than all of domains 21. Interpreter 20 alsoproduces first scores 68 1-N. Each of first scores 68 corresponds to oneof interpretations 67. In some aspects, scoring is performed prior tocompleting interpretations of expression 66. Scoring prior tointerpretation can save processing resources. Interpreter 20 can sendfirst scores 68 1-N to rescore module 24.

Rescore module 24 receives first scores 68 1-N from interpreter 20 andcorresponding weights. The corresponding weights can be included inmetadata 63 and/or accessed from storage based on metadata 63. Rescoremodule 24 applies the weights to first scores 68 1-N to produce secondscores. Rescoring can include multiplying a first score by acorresponding weight (or a value 1 for scores that have no associatedweight) to produce a second score. Other rescoring mechanisms, such asaddition, adding to logarithms, threshold comparisons, filtering, andapplication of arbitrary functions can also be used.

Based on the second scores, multi-domain natural language understanding(NLU) interpreting server 13 can select an appropriate interpretation ofexpression 66 (e.g., an interpretation with the highest score, aninterpretation that exceeds a threshold for a corresponding domain,etc.) from among interpretations 67 1-N. Based on the selectedappropriate interpretation, multi-domain natural language understanding(NLU) interpreting server 13 can generate an appropriate response toaudio samples 62. An appropriate response can include causing the clientdevice to perform an operation, such as outputting text or audiblemessages, playing a song, playing a video, moving, etc.

FIG. 7 illustrates a further more detailed view of multi-domain naturallanguage understanding (NLU) interpreting server 13. Multi-domainnatural language understanding (NLU) interpreting server 13 can receiveexpression package 83 from a client device, such as, for example, fromany of smart speaker, 12, mobile phone 14, or automobile 15.Multi-domain natural language understanding (NLU) interpreting server 13forwards expression 84 and metadata 86 to natural language understandingmodules 70.

As depicted, natural language understanding modules 70 receivesexpression package 83, including expression 84 and metadata 86. Naturallanguage understanding modules 70 can include the functionality of anyof interpreter 20, weight selector 23, and rescore module 24. Based onexpression 84 and metadata 86, natural language understanding modules 70can produce interpretations 76 1-N (of expression 84) and correspondingsecond scores 77 1-N respectively for domains 21.

Natural language understanding modules 70 can send second scores 77 1-Nto domain chooser 72 and interpretations 76 1-N to intent productionmodule 73. Domain chooser 72 can receive second scores 77 1-N fromnatural language understanding modules 70. Based on second scores 771-N, domain chooser 72 can choose domain 21B as the domain appropriatefor expression 84. Domain chooser 72 can indicate domain 21B to intentproduction module 73. Intent production module 73 can receive theindication of domain 21B from domain chooser 72 and can receiveinterpretations 76 1-N from natural language understanding modules 70.

Based on the indication of domain 21B and interpretations 76 1-N, intentproduction module 73 can produce intent 79. Intent 79 can be ahypothesis with respect to a user's meaning for creating expression 84,such as, for example, to look-up information, to perform an action, etc.In one aspect, intent 79 is an appropriate data structure, such as, forexample, represented with JavaScript object notation (JSON) or a set ofweb application programming interface (API) request arguments. Intentproduction module 73 can send intent 79 to response computation module74.

Response computation module 74 can receive intent 79 from intentproduction module 73. Based on intent 79, response computation module 74can compute response 82 (a response to expression 84). Response 82 canindicate information is to be returned to a user, an action is to beperformed, etc. Response 82 can include text, audio, speech audio,marked up code for text-to-speech (TTS), or other appropriate forms.

Platform User Interfaces

Aspects of the invention can provide a platform for client developers toconfigure the availability of domains to their client users and toconfigure sets of rescoring weights for the available domains. Domainchoices and weights can be entered through a text-based interface, agraphical user interface (GUI), or using other methods. FIG. 8illustrates an example GUI configurator screen 81.

GUI configuration screen 81 includes a title “myClientProject” alongwith various columns for configuring domain information. GUIconfiguration screen 81 represents each domain as a row with columns forenabling/disabling the domain, a domain weight, a domain name (e.g.,Weather, Date/Time, Timer, Calendar, etc.), and a domain description(describing what the domain does). A scroll bar can be used to scroll upor down the list of domains.

Domains can be enabled and disabled by toggling a user-interface control(e.g., a checkbox) in the enable column. Interpreter 20 can interpret anexpression in domains that are enabled. Interpreter 20 does notinterpret the expression in domains that are disabled. Disabling domainscan conserve computing resources, avoid costs associated with moreexpensive domains, avoid access to domains associated with informationthat might lead users away from desired actions (e.g., purchases), andprevent generation of inappropriate responses that might degrade userexperience.

Weights can be entered into a text field in the weight column. GUIconfiguration screen 81 can allow weights to be entered for each domainor just for enabled domains. Weights can be floating point numbersappropriate for multiplicative rescoring, integers appropriate formultiplicative rescoring, integers representing exponents for rescoringvalues, text-based descriptions such as “low”, “medium”, and “high”, orrepresented in various other appropriate ways. In GUI configurationscreen 81 weights are expressed as values from 0.1 to 9.9. Rescoring caninclude multiplying a first score by a weight to compute a second score.First scores can be normalized so that weights provide systemappropriate preferences for each domain.

As depicted, GUI configuration screen 81 indicates higher weights forthe navigation and music domain but a lower weight for the Wikipediadomain. The depicted configuration of weights may be appropriate for anautomobile client with a navigation system and radio. Wikipedia, whichoffers miscellaneous general knowledge, is a domain that automobilesshould offer, but that is not frequently used by most passengers andtherefore has a lower weight.

Domain Ranks

Another approach to improving the likelihood that a client appropriatelyresponds to potentially ambiguous user queries is to rank domains.Domain ranking can be used separately from or in conjunction with domainweights. Domain weights and domain ranks are complementary ways ofindicating domain priorities associated with natural languageexpressions.

FIG. 9 illustrates a further additional more detailed view ofmulti-domain natural language understanding (NLU) interpreting server13. As depicted, natural language understanding modules 70 receivesexpression package 93, including expression 94 and ranks 96.Multi-domain natural language understanding (NLU) interpreting server 13can receive expression package 93 from a client device, such as, forexample, from any of smart speaker, 12, mobile phone 14, or automobile15. Multi-domain natural language understanding (NLU) interpretingserver 13 forwards expression 94 to natural language understandingmodules 70 and forwards ranks 96 to domain chooser 72.

As depicted, natural language understanding modules 70 receivesexpression 94. Based on expression 94, natural language understandingmodules 70 can produce interpretations 1-N of expression 94 (not shown)and corresponding scores 97 1-N respectively for domains 21. Scores 971-N can be first scores (e.g., when domain ranks are used separately) orsecond scores (e.g., when domain ranks are used in combination withdomain weights). Natural language understanding modules 70 can sendscores 97 1-N to domain chooser 72. Natural language understandingmodules 70 can send interpretations 1-N (not shown) to intent productionmodule 73.

Domain chooser 72 can receive scores 97 1-N from natural languageunderstanding modules 70. Domain chooser 72 can apply ranks 96 to scores97 1-N. Based on ranks 96, domain chooser 72 can choose domain 21C asthe domain appropriately aligned with expression 94. Domain chooser 72can indicate domain 21C to intent production module 73. Intentproduction module 73 can receive the indication of domain 21C fromdomain chooser 72 and can receive interpretations from natural languageunderstanding modules 70. Intent production module 73 can determine anintent, and response computation module 74 can use the intent togenerate a response to expression 94.

In some aspects, domain ranks are carried as metadata in an expressionpackage. In other aspects, domain ranks are stored, for example, indurable storage. In these other aspects, domain ranks carried asmetadata can be used to override, bias, or reweight stored ranks.

Thus, it may be that a client sends domain ranks to an NLU interpretingserver with an expression. For example, the following domain rankings:

“DomainRank”: [“Movies”, “Music”, “Map”]

can be sent along with an expression to favor a Movies domain over aMusic domain and favor the Movies domain and the Music domain over a Mapdomain.

In some embodiments, an NLU interpreting server computes a Boolean valuefor each of a number of domains for an expression. Per domain, the valuerepresents whether or not the domain can interpret the expression. Whenmultiple domains can interpret the expression, the NLU interpretingserver can choose a domain appearing earliest in the DomainRank list. Inthe above example, the Movies domain is chosen if in competition withMusic or Map and Music is chosen if in competition with Map.

In other embodiments, an NLU interpreting server computes a score foreach of a number of domains for an expression. Per domain, the scorerepresents how likely it is that the expression is appropriate for thedomain. The expression can be indicated as appropriate for a domain whena score for the domain is above a score threshold. When multiple domainshave a score above the threshold, the NLU interpreting server can choosea domain appearing earliest in the DomainRank list

It may be that different domains have different score thresholds. Forexample, the following domain rankings:

-   -   “DomainRank”: [“Movies”:0.5, “Music”:0.75, “Map”:0.6]        can be sent along with an expression. The domain rankings define        a threshold of 0.5 for movies, a threshold of 0.75 for music,        and a threshold of 0.6 for map. Thus, a score for the Movies        domain has to be greater than 0.5, a score for the Music domain        has to be greater than 0.75, and a score for the Map domain has        to be greater than 0.6.

Specifying a DomainRank list is optional. If not present, a domainchoice can be based on interpretation score. In some embodiments, theDomainRank list does not need to be a comprehensive list of enableddomains. Domains not specified in the DomainRank list have an effectiverank of being below the lowest ranked domain on the list.

A client device can specify an order of the DomainRank list based oncontext. For example, a client device, such as, for example, a mobilephone, can include multiple applications. When the client device isrunning a map application in the foreground, a DomainRank can be changedto prioritize a Map domain. For example, the following domain rankings:

-   -   “DomainRank”: [“Map”, “Movies”, “Music” ]        can be used when a map application is in the foreground at a        mobile phone.

Other Domain Indicators

In some aspects, a domain rank can have an indicator representing aplurality of domains, such as one with a reserved named OTHER. In someembodiments, an OTHER domain indication represents any domains not namedin the DomainRank list. Using an OTHER domain is useful to highlyprioritize a domain of interest when the domain of interest caninterpret an expression. For example, the following domain rankings:

-   -   “DomainRank”: [“CarControl”, “OTHER” ]        highly prioritizes a CarControl domain to provide an        interpretation for any expression that uses the word “car”.

However, such a configuration of a DomainRank list may also cause userfrustration when a user intends for a different domain to interpret apotentially ambiguous expression. For example, the CarControl domain maybe unable to satisfy a request to play music by a band named “The Cars”.Such an approach may be suited for preferring domains with a relativelylimited range of interpretations. The limitation can be furtheralleviated by indicating domain ranks with per-domain score thresholds.In such an embodiment, the CarControl domain threshold would be given ahigher score threshold and OTHER a lower score threshold. Thus, chancesare increased that other domains are chosen for user expressions if theuser phrases the expression expressly for the intended OTHER domain.

An OTHER domain indication may also be useful to provide for a certaindomain being a default. For example, the following domain rankings:

-   -   “DomainRank”: [“OTHER”, “Web” ]        prioritizes any domain that can interpret an expression to be        chosen, but if no other domain can interpret the expression, a        Web domain is chosen by default. A score threshold can also be        used for OTHER domains. The score threshold applies to domains        not expressly listed in a DomainRank list.

1:1 Domain Comparison Rules

Aspects allow for separate expressions of pairs of domain ranks ormultiple lists of relative domain ranks. This is useful if, for example,domain A is favored over domain C and domain B is favored over domain Cbut neither A nor B is favored over each other. In such a scenario, anordered list would create a false ranking preference between domains Aand B.

Client Stored Weights

In some aspects, a client stores weights in an array and transmits thecontents of the array as domain weight metadata (e.g., weights 41) alongwith an expression (e.g., expression 29). When an event is detected, thearray values can be updated. An event can be detected by execution of aninterrupt service routine in a processor or by a polling routine. Aweight update, in some cases, is a switch between two values based on astate of a Boolean condition. Updated weights in some cases are theresult of an integer or floating-point output of any function of ameasured value.

For example, in an automobile, if the windshield wipers are switched on,a weather domain weight is increased because users are more likely tomake weather-related expressions when the weather is rainy compared towhen it is not. In an automobile, a navigation domain weight is set highimmediately after the navigation system gives a driver instruction, andgradually tapers down to a steady state value over a period of severalseconds after that since users are more likely to make anavigation-related expression shortly after the navigation system givesan instruction.

As another example, a device can include a music-related app that can beopened or closed. When a music app opening event occurs, a music domainweight is increased. On the other hand, when a music app closing eventoccurs, the music domain weight is decreased (e.g., to a default value).

Physical Embodiments

Various embodiments are non-transitory computer readable media, servers,embedded internet-of-things (IoT) client devices controlled bysystem-on-chip (SoC) semiconductor devices, and method of the operationor usage of any of the above types of embodiments. FIG. 10A illustratesan example rotating magnetic disk non-transitory computer readablemedium 101. Rotating optical disks and other mechanically moving storagemedia are possible. FIG. 10B illustrates an example (e.g., Flash) randomaccess memory (RAM) chip non-transitory computer readable medium 102.Other non-moving storage media packaged with leads or solder balls arepossible. Various embodiments of non-transitory computer readable mediastore computer code comprising instructions that, if executed by one ormore computer processors would cause the one or more computer processorsto perform the described method embodiments. Some embodiments arecomputer programs. Some embodiments comprise multiple storage media thatwork in conjunction with each other

FIG. 11A illustrates solder ball side of a packaged SoC 110. Variouspackage shapes and sizes are possible for various chip implementations.Many embedded systems and IoT devices function as clients and arecontrolled by SoC devices. FIG. 11B illustrates an example architectureof a SoC 111. As depicted, SoC 111 includes a multicore computerprocessor (CPU) 112, a multicore graphics processor (GPU) 113, displayinterface 117, I/O interfaces 118, dynamic random access memory (DRAM)interface 115, Flash interface 116, and network interface 119. Thecomponents communicate with one another over network-on-chip 114.

Multicore computer processor (CPU) 112 and multicore graphics processor(GPU) core 113 can connect through DRAM interface 115 to an externalDRAM chip for volatile program and data storage. Multicore computerprocessor (CPU) 112 and multicore graphics processor (GPU) core 113 canalso use Flash Interface 116 to connect to an external non-volatileFlash RAM non-transitory computer readable medium for computer programstorage. Display interface 117 can provide for a display for showing aGUI and an input/output (I/O) interface 118 can provide an ability toconnect to various I/O interface devices, as needed for different clientdevices. Client devices can include universal serial bus (USB)peripherals, Bluetooth peripherals, touch screens, audio analog todigital converters for microphones, power amplifier devices for drivingspeakers, global navigation system receiver interfaces, light emittingdiode (LED) drivers, and interfaces to other sensors. Network interface119 can provide access to WiFi, 3G, 4G long-term evolution (LTE), 5G,and other wireless interface standard radios as well as ethernetconnection hardware.

FIG. 12A illustrates an example of a rack-based server 120. Rack-basedserver 120 can include an array of server blades, each connected to anetwork (e.g., the Internet). The server blades can include CPU and GPUprocessor chips, RAM chips, and network interface chips. FIG. 12Billustrates an architecture for an example server 121. Server 121comprises a multi-core CPU chip 122 and a multicore GPU chip 123 thatcommunicates through an interconnect 124 with a DRAM chip 125 forvolatile program and data storage and a network interface chip 126 forcommunicate with local and internet networks.

In general, described components can be connected to one another over(or be part of) a network, such as, for example, a personal area network(PAN), a local area network (LAN), a wide area network (WAN), and eventhe Internet. Accordingly, the described components as well as any otherconnected computer systems and their components, can create messagerelated data and exchange message related data (e.g., near fieldcommunication (NFC) payloads, Bluetooth packets, Internet Protocol (IP)datagrams and other higher layer protocols that utilize IP datagrams,such as, Transmission Control Protocol (TCP), Hypertext TransferProtocol (HTTP), Simple Mail Transfer Protocol (SMTP), etc.) over thenetwork.

Any of the described devices, servers, components, speakers, phones,automobiles, interpreters, selectors, modules, choosers, etc., can beimplemented on the hardware components described in FIGS. 10A-12B.

The foregoing description has been presented for the purposes ofillustration and description. It is not intended to be exhaustive or tolimit the invention to the precise form disclosed. Many modificationsand variations are possible in light of the above teaching. Further, itshould be noted that any or all of the aforementioned alternateembodiments may be used in any combination desired to form additionalhybrid embodiments of the invention.

Further, although specific embodiments of the invention have beendescribed and illustrated, the invention is not to be limited to thespecific forms or arrangements of parts so described and illustrated.The scope of the invention is to be defined by the claims appendedhereto, any future claims submitted here and in different applications,and their equivalents.

What is claimed:
 1. A method comprising: receiving an expression packagecontaining a natural language expression and metadata; interpreting thenatural language expression according to a first domain to produce afirst interpretation and a first score for the first domain;interpreting the natural language expression according to a seconddomain to produce a second interpretation and a first score for thesecond domain; using the metadata to identify a first weight for thefirst domain and to identify a second weight for the second domain;applying the first weight to the first score for the first domain tocompute a second score for the first domain; applying the second weightto the first score for the second domain to compute a second score forthe second domain; selecting the first interpretation as the appropriateinterpretation of the natural language expression in view of the secondscore for the first domain and the second score for the second domain;and computing a response to the natural language expression based onselection of the first interpretation.
 2. The method as recited in claim1, wherein using the metadata to identify a first weight for the firstdomain and to identify a second weight for the second domain comprisesidentifying weights based on a client ID.
 3. The method as recited inclaim 1, wherein using the metadata to identify a first weight for thefirst domain and to identify a second weight for the second domaincomprises identifying weights based on a vendor ID and a product ID. 4.The method as recited in claim 1, wherein selecting the firstinterpretation as the appropriate interpretation comprises selecting thefirst interpretation based on the second score for the first domainbeing higher than the second score for the second domain.
 5. The methodas recited in claim 1, wherein selecting the first interpretation as theappropriate interpretation comprises selecting the first interpretationbased on the second score for the first domain exceeding a threshold. 6.The method as recited in claim 1, wherein receiving an expressionpackage containing a natural language expression and metadata comprisesreceiving the expression package and metadata from a client device, themetadata indicating which application is running in the foreground atthe client device.
 7. The method as recited in claim 6, furthercomprising taking a computerized action with respect to the clientdevice to implement the computed response.
 8. The method of claim 1,further comprising: providing a configurable platform for naturallanguage understanding for various types of clients; and receiving, froma developer, instructions to apply the first weight to the first domainand the second weight to the second domain at the configurable platform.9. A method comprising: receiving an expression package containing anatural language expression, a first weight corresponding to a firstdomain, and a second weight corresponding to a second domain;interpreting the natural language expression according to the firstdomain to produce a first interpretation and a first score for the firstdomain; interpreting the natural language expression according to thesecond domain to produce a second interpretation and a first score forthe second domain; applying the first weight to the first score for thefirst domain to compute a second score for the first domain; applyingthe second weight to the first score for the second domain to compute asecond score for the second domain; selecting the first interpretationas the appropriate interpretation of the natural language expression inview of the second score for the first domain and the second score forthe second domain; and computing a response to the natural languageexpression based on selection of the first interpretation.
 10. Themethod of claim 9, wherein receiving an expression package comprisesreceiving an expression package wherein the first weight is tagged withan identifier for the first domain and the second weight is tagged withan identifier for the second domain.
 11. The method of claim 9, whereinreceiving an expression package comprises receiving an expressionpackage containing a natural language expression and an array of domainweights.
 12. The method as recited in claim 9, wherein selecting thefirst interpretation as the appropriate interpretation comprisesselecting the first interpretation based on the second score for thefirst domain being higher than the second score for the second domain.13. The method as recited in claim 9, wherein selecting the firstinterpretation as the appropriate interpretation comprises selecting thefirst interpretation based on the second score for the first domainexceeding a threshold.
 14. The method as recited in claim 9, whereinreceiving an expression package containing a natural language expressionand weights comprises receiving an expression package from a clientdevice, the client device selected from among: a smart speaker, a mobilephone, and an automobile.
 15. The method as recited in claim 14, furthercomprising taking a computerized action with respect to the clientdevice to implement the computed response.
 16. A method comprising:receiving a natural language expression; accessing a domain rank listindicating a preference for interpretations from a first domain relativeto interpretations from a second domain; interpreting the naturallanguage expression according to the first domain to produce a firstinterpretation and a first score for the first domain; interpreting thenatural language expression according to the second domain to produce asecond interpretation and a second score for the second domain;determining that the first score exceeds a first threshold, the secondscore exceeds a second threshold, and the second score is higher thanthe first score; selecting the first interpretation as the appropriateinterpretation of the natural language expression in view of thepreference in the rank list and based on the first score exceeding thefirst threshold and notwithstanding that the second score is higher thanthe first score; and computing a response to the natural languageexpression based on selection of the first interpretation.
 17. Themethod as recited in claim 16, wherein accessing a domain rank listcomprises receiving the domain rank list along with the natural languageexpression.
 18. The method as recited in claim 16, wherein receiving anatural language expression comprises receiving a natural languageexpression from a client device, the client device selected from among:a smart speaker, a mobile phone, and an automobile.
 19. The method asrecited in claim 18, further comprising taking a computerized actionwith respect to the client device to implement the computed response.20. The method of claim 16, further comprising: providing a configurableplatform for natural language understanding for various types ofclients; and receiving, from a developer, instructions to give thepreference to interpretations from the first domain relative tointerpretations from the second domain.